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Executive  Summary 


This  paper  reports  the  results  of  researeh  on  whether  ehanges  in  Department  of 
Defense  (DoD)  aequisition  policy  and  process  have  had  a  discernible  effect  on  growth  of 
Program  Acquisition  Unit  Cost  (PAUC)  of  major  defense  acquisition  programs 
(MDAPs).  The  analysis  is  interesting  primarily  because  it  sheds  some  light  on  and — as  it 
turns  out — challenges  important  assumptions  that  often  are  implicit  in  discussions  of 
acquisition  reform. 

Acquisition  Regime  and  PAUC  Growth 

DoD  acquisition  policy  and  process  over  the  period  Fiscal  Year  (FY)  1970-FY  2007 
can  be  grouped  into  five  successive  regimes: 

1.  The  Defense  Systems  Acquisition  Review  Council  (DSARC),  1970-1982 

2.  The  Post-Carlucci  Initiatives  DSARC,  1983-1989 

3.  The  Defense  Acquisition  Board  (DAB),  1990-1993 

4.  Acquisition  Reform  (AR),  1994-2000 

5.  The  DAB  -  Post  Acquisition  Reform,  2001-2007 

The  table  on  page  iv  displays  the  average  PAUC  growth  for  MDAPs  that  passed 
Milestone  (MS)  II/B  or  filed  a  first  Selected  Acquisition  Report  (SAR)  in  each  of  these 
regimes.  The  PAUC  growth  figures  all  are  measured  from  the  MS  II/B  baseline  and 
normalized  to  the  MS  II/B  total  inventory  objective.  There  are  a  number  of  interesting 
aspects  to  these  data;  for  example,  the  high  PAUC  growth  during  the  AR  period  and  the 
lower  PAUC  growth  for  FY  2001-FY  2007.  Granting  that,  the  single  most  notable 
feature  of  these  data  is  the  absence  of  any  trend  in  PAUC  growth.  If  changes  in 
acquisition  policy  and  process  have  had  a  sustained  influence  on  PAUC  growth,  it  does 
not  show  up  in  this  table. 
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Average  PAUC  Growth  in  Successive  Acquisition  Regimes 


Acquisition  Regime 

Time  Period 

Average  PAUC 
Growth 

No.  of 

Observations 

DSARC 

1970-1982 

32% 

48 

Post  Carlucci  Initiatives  DSARC 

1983-1989 

19% 

40 

DAB 

1990-1993 

36% 

11 

Acquisition  Reform  (AR) 

1994-2000 

66% 

27 

DAB  post  AR 

2001-2007 

19% 

25 

Broadly,  there  are  two  ways  to  explain  the  absenee  of  sustained  effeets  of 
aequisition  poliey  and  proeess  on  the  PAUC  growth  data.  First,  they  may  in  faet  not  have 
a  strong  or  eonsistent  effeet  on  PAUC  growth.  Seeond,  aequisition  poliey  and  proeess 
may  have  substantial  effeets  that  are  masked  by  some  other  faetor  or  faetors. 

Funding  Climate  and  PAUC  Growth 

Thinking  along  the  lines  of  the  seeond  of  these  possibilities  led  to  eonsideration  of 
whether  ehanges  in  the  DoD  funding  elimate  might  be  assoeiated  with  PAUC  growth. 
The  period  1970-2007  ineludes  two  sub-periods  during  whieh  aequisition  funding  was 
Relatively  Constrained:  FY  1970-FY  1980  and  FY  1987-FY  2002.  It  also  ineludes  two 
sub-periods  in  whieh  the  aequisition  funding  elimate  was  more  aeeommodating: 
FY  1981-FY  1986  and  FY  2003-FY  2007.  The  following  table  displays  the  average 
PAUC  growth  data  for  these  four  sub-periods. 


Average  PAUC  Growth  under  Different  DoD  Topline  Conditions 


Relatively  Constrained 

Relatively  Accommodating 

Period  (FY) 

PAUC  Growth 

Period  (FY) 

PAUC  Growth 

1970-1980 

35%  (42) 

1981-1986 

12%  (35) 

1987-2002 

53%  (55) 

2003-2007 

7%  (19) 

Note:  Numbers  in  parentheses  are  the  number  of  observations  available. 


These  data  make  it  elear  that  MDAPs  that  passed  MS  II/B  in  Relatively  Constrained 
funding  elimates  had  far  larger  PAUC  growth  than  those  that  passed  MS  II/B  in  periods 
when  the  funding  elimate  was  Relatively  Aeeommodating — by  a  faetor  of  three  in  the 
first  eomparison  and  by  a  faetor  of  five  in  the  seeond. 

Acquisition  Regime  and  Funding  Climate 

The  table  on  page  v  expands  the  table  above  by  replaeing  the  funding  elimate  sub¬ 
periods  with  the  aequisition  poliey  and  proeess  regimes.  This  table  provides  results  for 
two  sets  of  natural  experiments.  First,  the  PAUC  growth  eolumns  give  the  effeet  of 
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changes  in  the  aequisition  regime  for  a  given  funding  elimate.  Second,  the  rows  show  the 
effeet  of  funding  elimate  for  a  given  aequisition  regime.  For  example,  the  first  eleven 
years  of  the  DSARC  (FY  1970-FY  1980)  were  in  a  Relatively  Constrained  funding 
elimate,  while  the  next  two  (FY  1981-FY  1982)  were  in  a  period  in  whieh  the  elimate  for 
aequisition  funding  was  Relatively  Aeeommodating. 


Average  PAUC  Growth  by  Acquisition  Regime  and  Funding  Ciimate 


Acquisition  Regime 

Reiativeiy  Constrained 

Reiativeiy  Accommodating 

Period  (FY) 

PAUC  Growth 

Period  (FY) 

PAUC  Growth 

DSARC 

1970-1980 

35%  (42) 

1981-1982 

11%  (6) 

Post  Carlucci  DSARC 

1987-1989 

34%  (11) 

1983-1986 

13%  (29) 

DAB 

1990-1993 

36%  (11) 

None 

N/A 

Acquisition  Reform  (AR) 

1994-2000 

66%  (27) 

None 

N/A 

DAB  post  AR 

2001-2002 

57%  (6) 

2003-2007 

7%  (19) 

Note'.  Numbers  in  parentheses  are  the  number  of  observations  available. 


Statistieal  analysis  of  the  data  behind  the  averages  in  this  table  leads  to  two 
eonelusions.  First,  there  is  no  statistieally  signifieant  improvement  or  worsening  of 
PAUC  growth  eorrelated  with  the  different  aequisition  poliey  regimes.  This  is  obvious 
for  the  Relatively  Aeeommodating  elimate  (eolumn  on  the  right).  In  eontrast,  PAUC 
growth  over  FY  1994-FY  2000  and  in  FY  2001-FY  2002  (eolumn  on  the  left)  is 
noticeably  higher  than  the  averages  for  previous  periods,  but  the  differenees  proved  not  to 
be  statistieally  signifieant  beeause  of  the  large  varianee  among  programs  in  each  period. 

Seeond,  PAUC  growth  tends  to  be  substantially  higher  in  a  Relatively  Constrained 
funding  climate  than  in  the  Relatively  Aeeommodating  elimate.  We  have  only  three 
natural  experiments  of  ehanges  in  funding  elimate  for  a  given  aequisition  regime,  sinee 
two  of  the  five  aequisition  regimes  (DAB  and  AR)  fall  entirely  within  one  funding 
elimate.  Each  of  these  three  natural  experiments  on  the  effeet  of  funding  climate  has  the 
same  outeome — MDAPs  that  passed  MS  II/B  in  a  Relatively  Constrained  funding  elimate 
on  average  have  a  mueh  higher  PAUC  growth  rate  than  those  that  passed  MS  II/B  in  a 
Relatively  Aeeommodating  funding  elimate  for  a  given  aequisition  regime.  The  outcomes 
of  the  first  two  experiments  are  virtually  identieal — an  average  PAUC  growth  of  35  and 
34  pereent,  respeetively,  in  the  two  periods  when  the  topline  was  Relatively  Constrained 
and  average  PAUC  growth  of  11  pereent  and  13  pereent,  respeetively,  in  the  two  periods 
when  the  topline  was  Relatively  Aeeommodating.  The  effeet  is  most  pronouneed  in  the 
third  experiment  (DAB  post-AR),  whieh  is  statistieally  signifieant  at  the  1  pereent 
eonfidenee  level,  as  are  the  differenees  for  the  DSARC  and  Post-Carlueci  DSARC 
regimes. 
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Does  the  Resource  Allocation  Process  Play  a  Major  Role  in  PAUC 
Growth? 

These  conclusions  tend  to  challenge  a  fundamental  assumption  implicit  in  most 
discussions  of  acquisition  reform:  that  the  main,  although  not  the  only,  causes  of  PAUC 
growth  are  to  be  found  in  the  acquisition  realm — the  effectiveness  of  the  Program 
Manager,  the  adequacy  of  the  developmental  test  plan,  the  reasonableness  of  the  cost 
estimate,  the  completeness  of  the  systems  engineering  plan,  among  many  others.  This 
assumption  is  hard  to  maintain  when  the  many  changes  in  acquisition  policy  and  process 
made  in  the  past  four  decades  have  not  had  statistically  significant  effects  on  PAUC 
growth,  but  there  is  a  statistically  significant  association  between  PAUC  growth  and 
funding  climate. 

The  association  between  PAUC  growth  and  funding  climate  suggests  that  the 
resource  allocation  process,  particularly  at  the  Service  level,  plays  an  important  role  in 
cost  growth.  This  means  more  than  “funding  instability.”  Funding  instability  is  a  term  of 
art  for  changes  in  MDAP  funding  through  the  annual  resourcing  cycle  and  “taxes.” 
Funding  instability  is  a  chronic  condition,  present  to  some  degree  in  all  periods.  What  this 
paper  observes  is  a  recurring  pattern — that  MDAPs  that  passed  MS  IFB  during  periods 
when  the  DoD  topline  was  Relatively  Constrained,  on  average,  had  much  higher  PAUC 
growth  than  those  that  passed  MS  II/B  during  a  period  of  Relatively  Accommodating 
funding  climate. 

The  conjecture  that  the  resource  allocation  process  plays  an  important  role  in  cost 
growth  gets  some  support  from  an  unexpected  direction — MDAPs  with  negative  cost 
growth,  of  which  there  are  twenty-nine  in  our  sample.  Negative  PAUC  growth  is 
recorded  if  the  actual  cost  of  a  program  proves  to  be  less  than  the  cost  in  the  MS  II/B 
baseline.  Assuming  the  program  was  funded  to  its  MS  II/B  baseline,  this  implies  that  over 
time  funds  can  be  taken  from  the  program  in  question  and  reallocated  to  other 
applications,  including  other  acquisition  programs.  The  program,  then,  effectively  can  be 
used  as  a  “bank” — a  way  to  hold  reserves  in  relative  safety  until  they  are  needed.  A  bank 
of  this  sort  is  more  likely  to  be  needed  in  a  Relatively  Accommodating  funding  climate, 
as  it  can  then  serve  as  a  way  to  delay  final  decisions  on  the  higher  level  of  funding  that 
has  become  available.  We  would  therefore  expect  to  find  relatively  more  instances  of 
negative  PAUC  growth  in  the  Relatively  Accommodating  funding  periods,  and  this  is 
what  we  observe.  About  30  percent  of  our  observations  in  Relatively  Accommodating 
funding  climates  are  of  negative  PAUCs,  compared  to  about  12  percent  across  the  periods 
of  Relatively  Constrained  climate. 

MDAPs  with  “high  cost  growth,”  which  we  define  as  quantity  normalized  PAUC 
growth  of  at  least  50  percent,  also  suggest  an  influence  from  resource  considerations. 
DoD  resource  managers,  particularly  at  the  Service  level,  have  only  a  few  tools  for 
responding  to  a  Relatively  Constrained  funding  climate.  One  of  these  is  to  impose  top- 
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down  limits  on  the  funding  for  particular  MDAPs  as  they  approach  MS  II/B.  The  result  is 
likely  to  be  particularly  optimistic  programmatic  and  costing  assumptions,  which  leads  to 
an  expectation  that  relatively  more  high  cost  growth  programs  will  be  observed  in  periods 
of  Relatively  Constrained  funding  climate.  This  is  what  we  observe.  During  periods  of 
Relatively  Constrained  funding  climate,  about  40  percent  of  MDAPs  had  very  high 
PAUC  growth.  In  contrast,  during  periods  of  Relatively  Accommodating  funding  climate 
only  about  7  percent  of  MDAPs  experienced  high  PAUC  growth. 

Taking  both  funding  climates  together,  85  percent  of  MDAPs  with  PAUC  growth  of 
at  least  50  percent  passed  MS  II/B  during  a  Relatively  Constrained  funding  climate. 
These  MDAPs  had  an  average  PAUC  growth  of  93  percent  and  accounted  for  just  over 
three-quarters  of  total  PAUC  growth.  Excluding  high  cost  growth  MDAPs  and  MDAPs 
with  negative  PAUC  growth,  average  PAUC  growth  across  the  two  funding  climates  was 
just  18  percent.  High  PAUC  growth  is  then  predominantly  a  feature  of  programs  with 
PAUC  growth  of  at  least  50  percent,  and  these  programs  mainly  passed  MS  II/B  in 
periods  of  Relatively  Constrained  funding  climates.  These  points  are  important  because 
they  suggest  that  reforms  directed  to  the  average  or  typical  MDAP  may  miss  the  real 
source  of  the  problem. 

Implications  for  Discussions  of  Acquisition  Reform 

This  paper  points  to  three  implications  for  a  discussion  of  acquisition  reform.  First, 
the  relevant  context  for  understanding  PAUC  growth  is  the  interface  between  the 
acquisition  process  and  the  resource  allocation  process.  The  crucial  evidence  behind  this 
point  is  the  strong  association  between  funding  climate  and  PAUC  growth.  Resource 
managers  must  think  in  terms  of  a  portfolio  of  programs  across  mission  areas  and 
commodity  types,  and  extending  from  efforts  in  the  technology  base  through  programs 
nearing  the  end  of  production.  When  a  program  is  completed,  it  opens  a  resource  “hole” 
that  programs  emerging  from  Engineering  and  Manufacturing  Development  can  occupy. 
In  turn,  programs  earlier  in  the  acquisition  cycle  can  move  forward  as  well.  When 
funding  for  acquisition  turns  down,  these  holes  get  smaller,  or  close  entirely,  or  require 
cuts  in  funding  for  ongoing  programs.  The  alternatives  available  in  this  circumstance  are 
all  undesirable — cancellations  of  programs,  delays  in  new  starts,  stretches,  and  adoption 
of  unrealistic  cost  estimates.  The  evidence  suggests  that  it  is  in  this  context  that  high 
PAUC  growth  arises. 

Second,  it  seems  unlikely  that  further  broad  changes  in  the  acquisition  process 
would  have  a  major  effect  on  PAUC  growth.  The  research  found  no  evidence  that  the 
efforts  to  strengthen  the  acquisition  process  through  the  years  have  resulted  in  lower  or 
higher  PAUC  growth.  This  does  not  mean  that  the  DAB  process  does  not  provide  a  useful 
discipline  on  acquisition  programs;  moreover,  further  changes  in  acquisition  policy  or 
process  might  be  warranted  for  reasons  of  good  government.  The  evidence  does,  at  a 
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minimum,  suggest  that  the  effeets  of  ehanges  in  the  aequisition  proeess  sinee  its 
inauguration  in  the  early  1970s  have  not  had  a  dominant  effeet  on  PAUC  growth. 

Third,  it  is  diffieult  to  see  that  the  eultures  of  the  DoD  aequisition  organizations  are 
a  erueial  obstaele  to  improved  performanee  on  eost  growth.  The  key  point  to  note  is  that 
high  PAUC  growth  is  not  persistent,  but  rather  episodie,  and  eorrelated  with 
environmental  faetors  outside  of  the  eontrol  of  the  aequisition  proeess.  There  is 
remarkably  little  PAUC  growth  in  periods  when  the  funding  is  Relatively 
Aeeommodating.  It  seems  fair  to  ask  if  it  makes  sense  to  assert  that  an  entrenched  culture 
sometimes  results  in  high  cost  growth  and  other  times  in  low  cost  growth.  Just  how  is  it 
that  the  A  team  takes  the  field  so  quickly  when  the  budgetary  sun  comes  out?  And  why, 
even  in  bad  budgetary  weather,  do  more  than  half  of  MDAPs  exhibit  comparatively 
modest  PAUC  growth? 
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A.  Introduction 

This  paper  reports  the  results  of  researeh  on  whether  ehanges  in  Department  of 
Defense  (DoD)  aequisition  poliey  and  proeess  have  had  a  diseemible  effeet  on  growth  of 
Program  Aequisition  Unit  Cost  (PAUC)  of  major  defense  aequisition  programs 
(MDAPs).  A  few  previous  studies  have  broaehed  this  topie,  but  it  has  reeeived  little 
attention,  and  the  work  that  has  been  done  has  not  resulted  in  any  aeeepted  or  even 
widely  reeognized  eonelusions/ 

The  establishment  of  the  Defense  Systems  Aequisition  Review  Couneil  (DSARC)  in 
late  Fiseal  Year  (FY)  1969  marks  the  start  of  systematie  Offiee  of  the  Seeretary  of 
Defense  (OSD)  oversight  of  MDAPs.  While  an  initiative  of  then  Deputy  Seeretary  of 
Defense  David  Paekard,  the  DSARC  responded  to  intense  eongressional  eoneerns  with 
growth  in  the  eosts  of  major  DoD  weapon  system  aequisition  programs.  It  appears  to 
have  generally  been  regarded  at  the  time  as  a  sueeessful  innovation. 

There  have  been  many  ehanges,  large  and  small,  in  DoD  aequisition  poliey  and 
proeess  sinee  the  DSARC  was  established.  Many  of  these  were  undertaken  simply  for 
reasons  of  good  government — to  reduee  the  eosts  of  the  deeision-making  proeess  and  the 
time  it  requires,  to  inerease  its  transpareney,  to  make  it  more  responsive  to  poliey 
direetion,  and  to  adapt  it  to  ehanges  in  the  teehnologieal  and  national  seeurity 
environment.  Many  others  were  aimed  direetly  at  improving  outeomes  on  MDAPs — in 
partieular,  redueing  eost  growth.  This  study  was  undertaken  in  the  hope  that  a  better 
understanding  of  the  effeets  of  these  ehanges  in  aequisition  poliey  and  proeess  on  PAUC 
growth  will  eontribute  to  the  long-running  diseussion  of  reform  of  the  DoD  aequisition 
proeess. 

Finding  or  making  estimates  of  PAUC  growth  for  a  suffieiently  large  set  of  MDAPs 
was  the  first  major  ehallenge  faeed  by  this  study.  We  have  a  PAUC  growth  estimate  for 


The  literature  includes  many  analyses  of  particular  acquisition  policies.  There  also  have  been  several 
largely  qualitative  studies  of,  for  example,  the  extent  to  which  a  set  of  related  initiatives  has  been 
successful  or,  to  offer  another  example,  the  apparent  successes  and  shortcomings  of  the  acquisition 
process  over  a  specific  time  period,  such  as  a  decade.  In  contrast,  there  have  been  few  broad 
quantitative  assessments  of  the  effectiveness  of  acquisition  policy  and  process.  The  main  predecessors 
of  this  work,  in  particular,  are  David  L.  McNicol,  Cost  Growth  in  Major  Weapon  Procurement 
Programs,  2nd  edition  (Alexandria,  VA:  Institute  for  Defense  Analyses,  2005),  especially  pages  41-44 
and  55-59,  and  Tyson  et  al.,  “The  Effects  of  Management  Initiatives  on  the  Costs  and  Schedules  of 
Defense  Acquisition  Programs,  Vol.  I  Main  Report,”  IDA  Paper  P-2722  (Alexandria,  VA:  Institute  for 
Defense  Analyses,  1992).  More  recently,  Obaid  Younossi  et  al.,  in  Is  Weapon  System  Cost  Growth 
Increasing?  A  Quantitative  Assessment  of  Completed  and  Ongoing  Programs  (Santa  Monica,  CA:  The 
RAND  Corporation,  2007),  concluded  from  their  careful  study  of  trends  in  development  cost  growth 
that  “despite  the  many  acquisition  reform  and  other  DoD  management  initiatives  over  the  years,  the 
development  cost  growth  of  military  systems  has  not  been  reduced.”  (Summary  page  xx)  References  to 
the  previous  literature  are  provided  in  both  McNicol  and  Younossi. 
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151  of  the  309  distinct  MDAPs  that  filed  at  least  one  SAR  during  FY  1969-  FY  2007,  a 
bit  less  than  half  of  the  total.  The  programs  for  which  we  have  a  PAUC  growth  estimate 
do  not  include  any  of  the  approximately  seventy-five  MDAPs  that  were  terminated  with 
little  or  no  production.  It  would  be  interesting  to  have  a  reasonable  PAUC  growth 
estimate  relative  to  the  Milestone  (MS)  11/B  baseline  for  these  programs,  but  developing 
such  estimates  would  require  far  more  resources  than  were  available  for  this  study.  The 
study,  then,  does  not  provide  a  comprehensive  picture  of  cost  growth;  doing  so  was  not 
its  intent.  The  question  asked  is  whether  changes  in  acquisition  policy  and  process  over 
time  have  visibly  had  an  influence  on  PAUC  growth.  We  ask  that  question  for  MDAPs 
that  passed  MS  11/B  as  Acquisition  Category  (ACAT)  1  programs  and  progressed  into  full 
rate  production.  The  question  is  not  explored  for  programs  that  were  cancelled  or 
truncated. 

Appendix  A  describes  the  sources  of  our  PAUC  growth  estimates  and  puts  the 
MDAPs  for  which  we  have  a  PAUC  growth  estimate  in  the  context  of  the  entire 
population  of  MDAPs.  The  data  we  used  are  included  on  a  compact  disc  (CD)  in  a  pocket 
on  the  inside  back  cover  of  this  report.  Unless  stated  otherwise,  PAUC  growth  here 
means  PAUC  growth  normalized  to  the  MS  11/B  baseline  quantity. 

The  second  major  challenge  was  one  of  research  design.  In  broad  outline,  the  paper 
identifies  natural  experiments  that  may  shed  some  light  on  the  effects  of  acquisition 
policy  on  PAUC  growth  and  then  interprets  the  outcomes  of  those  experiments.  In  part, 
this  is  straightforward.  We  know  when  the  main  changes  in  acquisition  policy  and 
process  occurred  and  what  they  were.  The  overall  DoD  acquisition  funding  climate  in 
various  periods — the  second  main  element  of  the  natural  experiment — also  can  be  readily 
established.  The  problem  is  that,  in  addition  to  the  easily  identified  elements  of  the 
natural  experimental  design,  there  are  a  considerable  number  of  other  factors  that  had 
some  influence  on  PAUC  growth.  We  first  limit  attention  to  acquisition  regime  and 
funding  climate  and  then,  as  particular  results  are  stated,  ask  whether  they  are 
compromised  by  the  omission  of  other  factors. 

B.  Building  Blocks 

Discussions  of  acquisition  reform  over  the  past  twenty-five  years  have  usually  put 
DoD  Program  Manager  (PM)  and  personnel  in  the  program  office  in  the  foreground. 
These  people  oversee  the  contractors  and  do  a  myriad  of  things  that  must  be  done  by  the 
government  for  a  major  acquisition  program  to  move  forward — contracting,  financial 
management,  and  test  planning,  among  many  others.  In  the  background  are  the 
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DoDI  5000.2  issued  Oct.  23,  2000,  formally  established  Milestones  A,  B,  and  C  (in  place  of  Milestones 
I,  II,  and  III)  as  the  main  decision  points  for  an  MDAP.  Milestones  A,  B,  and  C  began  to  be  used 
somewhat  earlier  for  new  programs,  however. 
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contractors  who  typically  do  the  development  and  manufaeturing.  A  good  program  will 
not  oeeur  if  the  government  personnel  and  eontraetors  do  not  do  their  jobs  well.  It  is 
equally  true  that  if  these  individuals  and  organizations  do  their  jobs  well,  a  good  outeome 
for  the  program  is  more  likely. 

What  this  foeus  on  the  DoD  PM,  the  program  offiee  personnel,  and  the  eontraetors’ 
PMs  and  workers  leaves  out  are  faetors  they  must  aeeept  as  “givens.”  These  givens  are 
subjeet  to  ehanges — sometimes  large  and  fairly  sudden — that  presumably  have 
substantial  eonsequenees  for  program  outeomes.  One  of  the  givens  is  the  topline  DoD 
funding  eonstraint,  whieh  does  not  determine,  but  generally  has  a  marked  influenee  on, 
the  funding  for  individual  MDAPs.  A  seeond  is  DoD  aequisition  poliey  and  proeess.  We 
begin  with  the  latter. 

1.  Acquisition  Regimes 

This  paper  distinguishes  five  sueeessive  DoD  aequisition  regimes: 

1 .  The  Defense  Systems  Aequisition  Review  Couneil  (DSARC),  1970-1982 

2.  The  Post-Carlueei  Initiatives  DSARC,  1983-1989"^ 

3.  The  Defense  Aequisition  Board  (DAB),  1990-1993 

4.  Aequisition  Reform  (AR),  1994-2000 

5.  The  DAB  -  Post  Aequisition  Reform,  2001-2007  (beeause  our  PAUC 
growth  data  ends  in  2007) 

The  transition  from  the  first  phase  of  the  DSARC  (1970-1982)  to  the  seeond  was 
prineipally  a  matter  of  poliey  direetion  and  renewal.  The  thirty-four  Carlueei  Initiatives 
(regime  number  2)  were  intended  to  inerease  the  effieieney  and  effeetiveness  of  the  OSD 
aequisition  oversight  proeess  and  the  Planning,  Programming,  and  Budgeting  System 
(PPBS).  While  the  DAB  (number  3)  itself  bears  a  strong  family  resemblanee  to  the 
DSARC,  the  statute  ereating  it  direeted  management  ehanges  intended  to  strengthen  what 
is  now  the  Under  Seeretary  of  Defense  (Aequisition,  Teehnology,  and  Logisties) 
(USD(AT&L)).  This  statute  also  ereated  the  position  of  Viee  Chairman  of  the  Joint 
Chiefs  of  Staff  (VCJCS)  and  direeted  a  new  requirements  proeess  eentered  on  the 


The  main  reference  we  have  used  is  J.  Ronald  Fox,  Defense  Acquisition  Reform,  1960  to  2009:  An 
Elusive  Goal  (Washington,  DC:  U.S.  Army  Center  of  Military  History,  2011).  Fox  identifies  the  main 
features  of  each  of  these  periods  as  well  as  the  most  important  changes  that  took  place  within  them. 

^  After  then  Deputy  Secretary  of  Defense  Frank  Carlueei.  There  is  some  uncertainty  about  when  the  Post- 
Carlucci  Reforms  DSARC  should  end  and  the  DAB  regime  should  begin.  The  relevant  statutes  were 
passed  in  1986,  and  the  DAB  began  functioning  under  that  name  in  late  FY  1987  or  early  FY  1988; 
however,  DoD  did  not  implement  the  full  set  of  reforms  required  by  statute  until  1990.  We  have  for  that 
reason  set  the  line  at  1990. 
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VCJCS.  The  results  sought  by  AR  (number  4)  were  improvements  in  MDAP  outcomes, 
but  changes  made  during  that  period  (not  all  part  of  AR)  somewhat  relaxed  OSD 
oversight  of  MDAPs.  There  were,  in  particular,  substantial  cuts  in  acquisition  staffs  at 
both  the  OSD  level  and  Service  Headquarters  level,  and  senior  decision  makers  took  a 
more  permissive  attitude  towards  cost  growth.  The  Post-AR  regime  (number  5)  was 
marked  by  the  arrival  of  a  new  administration  in  January  2001,  which  brought  policy 
changes  but  no  major  changes  to  the  acquisition  process  or  statutes. 

Table  1  displays  the  average  PAUC  growth  for  MDAPs  that  passed  MS  II/B  or  filed 
a  first  SAR  in  each  of  these  successive  regimes.  There  are  a  number  of  interesting  aspects 
to  these  data;  for  example,  the  high  PAUC  growth  during  the  AR  period  and  the  lower 
PAUC  growth  for  FY  2001-FY  2007.  Granting  that,  the  single  most  notable  feature  of 
these  data  is  the  absence  of  any  trend  in  PAUC  growth.  If  changes  in  acquisition  policy 
and  process  have  had  a  sustained  influence  on  PAUC  growth,  it  does  not  show  up  in  this 
table. 


Table  1.  Average  PAUC  Growth  in  Successive  Acquisition  Regimes 


Acquisition  Regime 

Time  Period 

Average  PAUC 
Growth 

No.  of 

Observations 

DSARC 

1970-1982 

32% 

48 

Post  Carlucci  Initiatives  DSARC 

1983-1989 

19% 

40 

DAB 

1990-1993 

36% 

11 

Acquisition  Reform  (AR) 

1994-2000 

66% 

27 

DAB  post  AR 

2001-2007 

19% 

25 

In  constructing  Table  1,  we  assigned  the  PAUC  growth  of  each  program  to  the 
acquisition  regime  in  place  when  the  program  passed  MS  IFB  or  filed  its  first  SAR.  At 
first  glance  this  may  seem  dubious,  since  a  program  can  easily  take  ten  or  fifteen  years 
from  the  start  of  Engineering  and  Manufacturing  Development  (EMD)  through  delivery 
of  the  final  production  lot  and  thus  spend  parts  of  its  acquisition  cycle  under  successive 
acquisition  regimes.  Note,  however,  that  our  estimates  are  of  PAUC  growth  measured 
from  the  baseline  established  at  MS  II/B,  which  does  not  change  over  the  course  of  a 
program’s  acquisition  cycle.  It  remains  possible  that  the  actual  acquisition  costs  of  a 
program  are  significantly  influenced  by  policy  or  process  changes  made  after  its  MS  II/B. 
Evidence  presented  in  Appendix  B  suggests  that  if  such  influences  exist,  they  are  much 
smaller  than  the  effect  of  the  cost  estimate  in  the  MS  II/B  baseline. 

Broadly,  there  are  two  ways  to  explain  the  absence  of  sustained  effects  of 
acquisition  policy  and  process  on  the  PAUC  growth  data,  hirst,  they  may  in  fact  not  have 
a  strong  or  consistent  effect  on  PAUC  growth.  Second,  acquisition  policy  and  process 
may  have  substantial  effects  that  are  masked  by  some  other  factor  or  factors. 
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2.  Funding  Climate 

Thinking  along  the  lines  of  the  seeond  of  these  possibilities  led  to  consideration  of 
whether  changes  in  the  DoD  acquisition  funding  climate  might  be  associated  with  PAUC 
growth.  We  have  a  PAUC  growth  estimate  for  151  MDAPs  that  passed  MS  II/B  or 
submitted  a  first  SAR  during  FY  1969-FY  2007.  This  period  includes  two  sub-periods 
during  which  acquisition  funding  was  Relatively  Constrained:  FY  1970-FY  1980  and 
FY  1987-FY  2002.  We  also  have  two  sub-periods  in  which  the  acquisition  funding 
climate  was  Relatively  Accommodating:  FY  1981-FY  1986  and  FY  2003-FY  2007.  (We 
did  not  include  any  program  not  at  least  five  years  beyond  MS  II/B,  and  the  2012  SARs 
were  the  last  available  for  this  study.)^ 

Table  2  displays  the  average  PAUC  growth  data  for  these  four  sub-periods.  The 
average  PAUC  growth  in  periods  of  Relatively  Constrained  acquisition  funding  is  far 
larger  than  it  is  in  periods  of  a  Relatively  Accommodating  funding  climate — by  a  factor 
of  three  in  the  first  comparison  and  by  a  factor  of  five  in  the  second. 


Table  2.  Average  PAUC  Growth  during  Different  Acquisition  Funding  Climates 


DoD  Topline  Relatively  Constrained 

DoD  Funding  Relatively  Accommodating 

Period  (FY) 

PAUC  Growth 

Period  (FY) 

PAUC  Growth 

1970-1980 

35%  (42) 

1981-1986 

12%  (35) 

1987-2002 

53%  (55) 

2003-2007 

7%  (19) 

Note:  Numbers  in  parentheses  are  the  number  of  observations  available. 


What  the  data  in  Table  2  portray  goes  well  beyond  “budget  instability”  as  usually 
understood.  Budget  instability  is  a  term  of  art  for  changes  in  MDAP  funding  through  the 
annual  resourcing  cycle  and  “taxes.”  Budget  instability  is  a  chronic  condition,  present  to 
some  degree  in  all  periods.  What  this  paper  observed  is  a  recurring  pattern — that  MDAPs 
that  passed  MS  II/B  during  periods  when  the  acquisition  funding  was  Relatively 
Constrained,  on  average,  had  much  higher  PAUC  growth  than  those  that  passed  MS  II/B 
during  periods  of  a  Relatively  Accommodating  funding  climate. 


We  use  as  the  breakpoints  that  define  these  periods  events  that  marked  major  changes  in  expectations 
about  the  course  of  defense  spending:  (1)  The  invasion  of  Afghanistan  by  the  Soviet  Union  in 
December  1979  (FY  1980),  which  about  a  month  later  led  President  Carter  to  announce  a  policy  of 
sustained  increases  in  defense  spending  starting  with  the  FY  1981  funding;  (2)  the  adoption  in 
December  1985  (end  of  the  first  quarter  of  FY  1986)  of  the  Gramm-Rudman-Hollings  Act,  the  funding 
constraints  of  which  effectively  ended  the  Carter-Reagan  defense  buildup;  and  (3)  the  9/1  l/2001attacks. 
It  is  important  to  recognize  that  key  decisions  made  within  DoD  on  content,  costing,  and  funding  for 
particular  MDAPs  in  a  given  year  are  made  at  least  a  year  in  advance  of  the  submission  of  the  funding 
for  that  year  to  the  Congress.  Consequently,  the  DoD  decisions  in  funding  submissions  reflect 
expectations  about  the  climate  that  submission  will  encounter. 
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C.  Statistical  Results 

Table  3  expands  Table  2  by  replaeing  the  funding  elimate  sub-periods  with  the 
aequisition  poliey  and  proeess  regimes.  This  table  provides  results  for  two  sets  of  natural 
experiments.  First,  the  PAUC  growth  eolumns  give  the  effeet  of  ehanges  in  the 
aequisition  regime  for  a  given  funding  elimate.  Seeond,  the  rows  show  the  effeet  of 
funding  elimate  for  a  given  aequisition  regime.  For  example,  the  first  eleven  years  of  the 
DSARC  (FY  1970-FY  1980)  were  in  a  tight  funding  elimate,  while  the  next  two 
(FY  1981-FY  1982)  were  in  a  period  in  whieh  the  aequisition  funding  elimate  was 
Relatively  Aeeommodating. 

The  two  seetions  that  follow  discuss,  in  turn,  whether  changes  in  acquisition  policy 
and  process  have  visible  effects  on  PAUC  growth,  and  the  association  between  funding 
climates  and  PAUC  growth. 


Table  3.  Average  PAUC  Growth  by  Acquisition  Regime  and  Funding  Climate 


Acquisition  Regime 

Relatively  Constrained 

Relatively  Accommodating 

Period  (FY) 

PAUC  Growth 

Period  (FY) 

PAUC  Growth 

DSARC 

1970-1980 

35%  (42) 

1981-1982 

11%  (6) 

Post  Carlucci  DSARC 

1987-1989 

34%  (11) 

1983-1986 

13%  (29) 

DAB 

1990-1993 

36%  (11) 

None 

N/A 

Acquisition  Reform  (AR) 

1994-2000 

66%  (27) 

None 

N/A 

DAB  post  AR 

2001-2002 

57%  (6) 

2003-2007 

7%  (19) 

Note:  Numbers  in  parentheses  are  the  number  of  observations  available. 


1,  Any  Trend  in  PAUC  Growth? 

There  is  no  doubt  that  direction  from  the  Milestone  Decision  Authority  (MDA) 
changes  particular  MDAPs,  and  some  of  those  changes  reduce  the  risks  of  major  PAUC 
growth  or  other  program  performance  shortfalls.  Viewed  from  this  perspective,  the 
question  asked  here  is  whether  the  decisions  made  (or  not  made)  in  different  acquisition 
regimes  are  large  enough  and  frequent  enough  to  be  visible  in  average  PAUC  growth. 

Looking  first  at  the  Relatively  Accommodating  climate  (column  on  the  right  in 
Table  3)  and  recognizing  that  it  is  likely  that  the  average  PAUC  growth  for  FY  2003- 
FY  2007  eventually  will  be  a  few  percentage  points  higher,^  we  can  see  no  trend  towards 


We  have  a  PAUC  growth  estimate  for  nineteen  of  the  twenty-five  MDAPs  that  passed  MS  B  as  ACAT  I 
programs  during  2003-2007  and  which  have  not  been  cancelled  or  truncated.  Of  these  nineteen 
programs,  six  have  been  completed,  six  are  in  full  rate  production  (FRP),  three  are  in  Low  Rate  Initial 
Production  (LRIP),  and  four  are  in  EMD.  Younossi  et  al.,  in  Is  Weapon  System  Cost  Growth 
Increasing?,  found  that  on  average,  60  percent  of  development  cost  growth  occurs  by  five  years  after 
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reduction  in  average  PAUC  growth  in  periods  with  a  Relatively  Accommodating  funding 
climate.  Statistical  analysis  is  consistent  with  this  impression;  that  is,  we  found  no 
evidence  of  statistically  significant  differences  among  average  PAUC  growth  rates  for  the 
Relatively  Accommodating  funding  climate.^ 

The  average  PAUC  growth  rates  for  the  two  most  recent  acquisition  regimes  during 
a  Relatively  Constrained  funding  (column  on  the  left  in  Table  3)  are  noticeably  larger 
than  those  for  the  three  earlier  periods.  Again,  however,  the  statistical  analysis  did  not 
indicate  that  any  of  the  averages  is  statistically  different  from  the  others  at  the  1  percent 
confidence  level. 

Appendix  C  presents  a  table  similar  to  Table  3  for  each  of  the  Military  Departments 
and  for  joint  programs.  None  shows  an  improving  trend  in  PAUC  growth  in  either  of  the 
two  funding  climates,  and  with  a  small  number  of  exceptions,  these  tables  show  the  same 
features  we  see  in  Table  3. 

We  have  no  fully  comparable  PAUC  growth  data  for  the  periods  before  the  DSARC 
was  established.  Consequently,  the  statistical  analysis  leaves  open  the  possibility  that  the 

g 

DSARC  and  its  successors  provided  a  useful  discipline  on  acquisition  programs. 
Moreover,  the  statistical  analysis  does  not  erase  history.  Weapon  system  cost  growth  was 
a  particular  concern  during  the  1980s — the  sixth  Carlucci  Initiative  was  “Funding  to  Most 
Likely  Costs” — and  it  is  reasonable  to  believe  that  the  Carlucci  Initiatives  did  in  fact  lead 
to  more  vigorous  enforcement  of  realistic  funding.  We  also  know  that  less  emphasis  was 
placed  on  weapon  system  cost  growth  during  the  AR  years,  and  oversight  of  acquisition 
programs  was  somewhat  more  relaxed.  It  could  be  that  these  differences  do  provide  part 
of  the  explanation  for  the  higher  observed  average  PAUC  growth  during  the  AR  years. 
The  statistical  analysis  prevents  us  from  asserting  with  confidence  that  they  do,  however. 


MS  B.  (31).  Estimates  of  procurement  cost  growth  also  usually  are  increased  as  necessary  to  reflect 
EMD  experience.  Since  each  of  the  nineteen  programs  is  at  least  five  years  beyond  MS  B,  even  a 
doubling  of  the  10  percent  average  PAUC  growth  would  be  unexpected. 

^  The  method  used  was  one-way  analysis  of  variance  (ANOVA).  ANOVA  is  a  test  of  whether  three  or 
more  samples  are  drawn  from  populations  with  the  same  mean.  The  null  hypothesis  is  that  all 
population  means  are  equal;  the  alternative  hypothesis  is  that  at  least  one  mean  is  different.  In  this  case, 
the  alternative  hypothesis  was  rejected  at  the  1  percent  level.  ANOVA  assumes  that  (1)  the  populations 
from  which  the  samples  were  drawn  are  normally  distributed,  (2)  the  samples  are  independent,  and  (3) 
the  variances  of  the  populations  are  equal.  We  are  grateful  to  Dr.  Sarah  Bums  for  her  advice  on  the 
statistical  analysis  and  for  doing  the  computations. 

g 

The  most  nearly  comparable  data  seems  to  be  that  in  Table  A-7  (pp.  A-6  to  A-8)  of  Appendix  A  of 
Tyson  et  al.,  “The  Effects  of  Management  Initiatives  on  the  Costs  and  Schedules  of  Defense 
Acquisition  Programs,  Vol.  I:  Main  Report.”  These  data  are  quantity  adjusted,  but  for  some  programs 
cost  growth  may  not  be  measured  from  the  estimates  at  the  start  of  EMD,  and  the  sample  may  include 
programs  that  were  cancelled.  Omitting  one  program  with  an  extremely  high  cost  growth  (Condor),  the 
average  PAUC  growth  for  the  pre-1970  MDAPs  was  48  percent,  which  is  noticeably  higher  than  the 
average  PAUC  growths  during  1970-1978. 
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because  those  differences  are  within  the  bounds  of  what  can  be  expected  from  the 
variability  of  the  data.^ 

It  remains  possible  that  factors  that  have  not  been  considered  in  this  paper  mask 
significant  influences  of  acquisition  policy  and  process  on  PAUC  growth  that  a  more 
refined  analysis  would  reveal.  In  considering  this  possibility,  it  must  be  kept  in  mind 
that  the  issue  is  growth  in  PAUC  from  the  MS  II/B  baseline  not  the  trend  over  time  in 
costs  for  a  commodity  group  or  the  acquisition  portfolio  as  a  whole.  Such  trends,  which 
may  well  exist,  do  not  necessarily  imply  more  cost  growth  for  individual  programs,  as 
they  should  be  reflected  in  the  MS  II/B  baselines.  Further  comments  on  possible 
confounding  variables  are  made  below.  The  conclusion  offered  here  is  that  once  we 
normalize  for  funding  climate  we  do  not  observe  any  improvement  in  PAUC  growth 
from  the  changes  made  over  the  years  in  acquisition  policy  and  process. 

2.  Association  of  Funding  Climate  and  PAUC  Growth 

Returning  to  Table  3  (on  page  6)  the  relevant  comparisons  are  between  the 
Relatively  Constrained  and  Relatively  Accommodating  funding  climates  for  a  given 
acquisition  regime.  Whereas  the  effects  on  PAUC  growth  of  the  different  acquisition 
regimes  are  elusive,  those  of  the  contrasting  funding  periods  stand  out  sharply. 

We  have  only  three  experiments  of  changes  in  funding  climate  for  a  given 
acquisition  regime,  since  two  of  the  five  acquisition  regimes  (DAB  and  AR)  fall  entirely 
within  one  funding  climate — Relatively  Constrained.  Each  of  these  three  natural 
experiments  on  the  effect  of  funding  climate  has  the  same  outcome — passing  MS  II/B 
during  a  Relatively  Constrained  funding  climate  is  on  average  associated  with  much 
higher  PAUC  growth  compared  to  passing  during  a  Relatively  Accommodating  funding 
climate  for  a  given  acquisition  regime.  The  outcomes  of  the  first  two  experiments  are 
virtually  identical — an  average  PAUC  growth  of  35  and  34  percent,  respectively,  in  the 
two  periods  when  the  topline  was  Relatively  Constrained  and  average  PAUC  growth  of 
11  percent  and  13  percent,  respectively,  in  the  two  periods  when  the  topline  was 


For  example,  the  exceptionally  high  average  PAUC  growth  during  the  AR  years  (66  percent)  can  be 
attributed  in  part  to  changes  in  the  program  mix.  During  the  AR  years,  four  helicopter  programs  passed 
MS  II/B,  one  more  than  average  for  a  period  of  this  length.  Moving  the  helicopter  program  with  the 
highest  PAUC  growth  (H-1  Upgrades)  from  1994-2000  to  1987-1989  reduces  the  average  PAUC 
growth  for  the  AR  years  from  66  percent  to  61  percent,  and  increases  the  average  for  1987-1989  from 
34  percent  to  48  percent. 

Results  of  McNicol,  Cost  Growth  in  Major  Weapon  Procurement  Programs,  2nd  edition,  point  to  one 
possibility — changes  in  programs  that  occur  after  they  pass  MS  II/B.  This  work  considered  growth  in 
quantity  normalized  unit  procurement  costs  after  excluding  costs  of  unforced  changes  in  program 
content  (i.e.,  changes  not  required  to  overcome  some  problem  “baked  into”  the  program  in  the  MS  II/B 
baseline).  McNicol  found  evidence  that  some  changes  in  the  acquisition  process  had  had  a  statistically 
significant  effect  on  this  measure  of  cost  growth.  See,  in  particular,  pp.  43-44  and  55-56. 


8 


Relatively  Accommodating.  The  effect  seems  most  pronounced  in  the  third  experiment 
(DAB  post-AR)— 57  percent  for  FY  2001-FY  2002  and  7  percent  for  FY  2003-FY  2007. 
(As  noted  earlier,  PAUC  growth  for  the  later  period  will  increase  somewhat  as  the 
programs  of  that  period  are  completed;  see  footnote  7  on  page  7.)  The  statistical  analysis 
found  the  each  of  these  differences  to  be  significant  at  more  than  the  1  percent  level. 

There  is  a  distinct  pattern  to  the  changes  in  funding  climate  over  our  sample 
period — ^bust,  boom,  bust,  boom.  If  some  other  factor  or  combination  of  factors  is 
actually  at  work,  rather  than  funding  climate,  it  would  have  to  have  this  same  pattern. 
One  possibility  is  that  the  methods  used  to  estimate  PAUC  growth  for  MDAPs  that 
passed  MS  II  during  FY  I989-FY  2007  are  not  the  same  as  the  method  used  for  those 
that  passed  MS  II  during  FY  1970-FY  1988.  The  limited  evidence  we  have  on  this 
possibility  is  presented  and  discussed  in  the  section  in  Appendix  A  entitled  “Comparison 
of  the  PA&E  and  CLC  PAUC  Growth  Estimates”  (page  A-7).  It  suggests  that  differences 
in  estimating  methods  do  not  explain  the  low  average  PAUC  growth  recorded  for 
EY  2003-EY  2007  or  the  comparatively  high  cost  growth  observed  for  EY  1990- 
EY  2002.  The  obvious  interpretation  of  Table  3,  and  the  one  we  believe  to  be  correct,  is 
that  it  really  is  changes  in  funding  climate  at  work. 

This  does  not  mean  that  a  Relatively  Constrained  funding  climate  causes  PAUC 
growth.  The  proximate  causes  of  PAUC  growth  are  decisions  embedded  in  programs 
approved  at  MS  II/B  (unrealistic  cost  estimates  or  programmatic  assumptions,  for 
example)  and  decisions  made  during  program  execution  (such  as  failing  to  act  promptly 
enough  on  test  results)  that  eventually  lead  to  PAUC  growth.  The  correlation  observed 
between  higher  PAUC  growth  and  periods  of  tighter  funding  climate  observed  does 
suggest  that  programs  are  more  likely  to  be  burdened  with  such  decisions  if  they  passed 
MS  II/B  during  a  Relatively  Constrained  funding  climate. 

D.  Is  High  PAUC  Growth  Systemic  in  the  Relatively  Constrained 
Funding  Climate? 

Based  on  the  analysis  thus  far,  it  would  not  be  surprising  to  find  that  almost  all 
programs  that  pass  MS  II/B  during  a  period  with  a  Relatively  Constrained  funding 
climate  are  burdened  with  the  sorts  of  very  optimistic  programmatic  and  costing 
assumptions  that  tend  to  result  in  high  PAUC  growth.  Alternatively,  the  bulk  of  the  cost 
growth  might  be  accounted  for  by  a  relatively  small  number  of  MDAPs.  Which  of  these 
two  cases  is  the  more  nearly  accurate  is  relevant  to  discussion  of  acquisition  reform.  In 
the  first  case  it  is  reasonable  to  assume  that  PAUC  growth  is  a  systemic  problem.  It  is 
often  said,  for  example,  that  the  acquisition  culture  has  a  bias  in  favor  of  optimistic 


In  this  case  we  used  the  usual  one-tail  test  for  the  difference  between  the  means  of  samples  drawn  from 
what  are  assumed  to  be  normal  populations. 
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programmatic  and  cost  assumptions.  PAUC  growth  looks  much  less  like  a  systemie 
problem  with  the  aequisition  proeess,  however,  if  most  of  it  is  due  to  a  small  number  of 
MDAPs. 

The  first  eolumn  of  Table  4  shows  the  average  PAUC  growth  (in  periods  of 
Relatively  Constrained  funding  climate)  of  MDAPs  that  had  a  PAUC  growth  between 
zero  and  50  pereent.  The  average  for  these  MDAPs  was  a  PAUC  growth  of  about  22 
pereent.  The  seeond  eolumn  shows  the  average  PAUC  growth  of  those  MDAPs  that 
experieneed  a  PAUC  growth  of  at  least  50  percent.  These  range  from  a  low  of  71  pereent 
(FY  1970-FY  1980)  to  a  high  of  122  pereent  (FY  1994-FY  2000).  The  average  of  these 
values  is  94  pereent.  Finally,  the  last  eolumn  in  Table  4  shows  the  percentage  of  PAUC 
growth  in  these  periods  aeeounted  for  by  MDAPs  with  PAUC  growth  of  at  least  50 
pereent.  The  range  is  62  pereent  to  89  pereent  and,  averaged  aeross  all  five  periods,  the 
high  cost  growth  MDAPs  accounted  for  just  over  three-quarters  of  total  PAUC  growth. 
(The  figures  shown  in  Table  4  are  eomputed  from  simple  averages  rather  than  weighted 
by  program  size.  )  In  short,  PAUC  growth  is  mainly  an  afflietion  of  Relatively 
Constrained  funding  elimates  and  it  is  primarily  due  to  a  minority  of  programs — on  the 
order  of  37  pereent — that  experienee  PAUC  growth  of  upwards  of  50  pereent. 


Table  4.  Characteristics  of  PAUC  Growth  in  Relatively  Constrained  Funding  Climate 


Acquisition  Regime 

Period 

(FY) 

Average  PAUC 
Growth  of 
MDAPs  with 
PAUC  Growth 
between 

0%  and  50% 

Average  PAUC 
Growth  of 
MDAPs  with 
PAUC  Growth 
>  50% 

%  of  PAUC 
Growth 
Accounted  for 
by  MDAPs  with 
PAUC  Growth 
>  50% 

DSARC 

1970-1980 

21%  (22) 

71%  (15) 

73% 

Post  Carlucci  DSARC 

1987-1989 

22%  (7) 

117%  (2) 

62% 

DAB 

1990-1993 

21%  (7) 

84%  (3) 

64% 

Acquisition  Reform 

1994-2000 

22%  (10) 

122%  (13) 

89% 

DAB  post  AR 

2001-2002 

29%  (3) 

85%  (3) 

75% 

Note'.  Numbers  in  parentheses  are  the  number  of  observations. 


Weighting  by  program  size  would  be  required  in  any  consideration  of  the  effect  of  PAUC  growth  on 
funding,  because  cost  growth  on  a  large  program  has  a  greater  effect  on  funding  requirements  than  cost 
growth  of  the  same  magnitude  on  a  smaller  program.  This  paper,  however,  is  concerned  with  examining 
the  extent  to  which  PAUC  growth  is  associated  with  particular  combinations  of  acquisition  regimes  and 
funding  climates  and  in  such  a  context,  each  observation  counts  as  much  as  any  other. 
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E.  Does  the  Resource  Allocation  Process  Play  a  Major  Role  in  PAUC 
Growth? 

This  section  turns  to  a  discussion  of  MDAPs  that  experienced  very  high  cost  growth 
(interpreted  as  a  PAUC  growth  of  at  least  50  percent)  and  MDAPs  that  experienced 
negative  cost  growth.  Investigation  of  this  topic  was  initially  prompted  by  the  prospect 
that  instances  of  high  cost  growth  and  perhaps  also  of  negative  cost  growth  mask  effects 
of  acquisition  policy  and  process  on  PAUC  growth.  The  topic  proves  to  be  interesting  for 
other  reasons  as  well.  First,  it  provides  clear  and  unexpected  evidence  of  the  connection 
between  PAUC  growth  and  funding  climate  and,  by  implication,  the  DoD  resource 
allocation  process.  Second,  cost  growth  proves  not  to  be  a  problem  with  the  typical 
system  but  with  the  minority  of  MDAPs  that  experience  very  high  cost  growth. 

1.  Negative  PAUC  Growth 

Twenty-nine  MDAPs  in  our  sample  show  negative  PAUC  growth  (not  including 
four  cases  of  zero  PAUC  growth).  Viewed  from  an  acquisition  perspective,  negative 
PAUC  growth  seems  anomalous;  in  fact,  it  is  not  uncommon  to  hear  confident  assertions 
to  the  effect  that  MDAPs  never  underrun  their  funding.  It  is  understandable,  however,  in 
a  resource  allocation  context  at  the  Service  level. 

Negative  PAUC  growth  is  recorded  if  the  actual  cost  of  a  program  proves  to  be  less 
than  the  cost  in  the  MS  II/B  baseline.  Negative  PAUC  growth  can  occur  because  a 
program  was  particularly  well  managed  or  lucky.  It  also  can  occur  if  the  ambitions  of  a 
program  are  scaled  back  after  a  program  has  passed  MS  II/B.  In  addition,  negative  PAUC 
growth  can  grow  out  of  resource  allocation  imperatives. 

Assuming  the  program  was  funded  to  its  MS  II/B  baseline,  negative  PAUC  growth 
implies  that  over  time  funds  can  be  taken  from  the  program  in  question  and  reallocated  to 
other  applications,  including  other  acquisition  programs.  The  program,  then,  effectively 
can  be  used  as  a  “bank” — a  way  to  hold  reserves  in  relative  safety  until  they  are  needed. 
A  “withdrawal”  can  be  made  in  the  execution  year  with  the  approval  of  the  Congress,  but 
for  the  outyears  of  the  Future  Years  Defense  Plan  (FYDP),  the  Service  can  simply  initiate 
the  reallocation  in  its  Program/Funding  submission  to  OSD. 

A  bank  of  this  sort  is  more  likely  to  be  needed  in  a  Relatively  Accommodating 
funding  climate,  as  it  can  then  serve  as  a  way  to  delay  final  decisions  on  allocation  of  the 
higher  level  of  funding  that  has  become  available.  We  would  therefore  expect  to  find 
relatively  more  instances  of  negative  PAUC  growth  in  the  Relatively  Accommodating 
funding  periods,  and  this  is  what  we  observe.  As  the  data  in  Table  5  indicate,  about  30 
percent  of  our  observations  in  Relatively  Accommodating  funding  climates  are  of 
negative  PAUCs,  compared  to  about  12  percent  across  the  periods  of  Relatively 
Constrained  climate. 
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Table  5.  Number  of  PAUC  Growth  Observations  less  than  Zero 
by  Acquisition  Regime  and  Funding  Climate 


Topline  Relatively 
Constrained 

Topline  Relatively 
Accommodating 

Acquisition  Regime 

Time  Period 
(FY) 

PAUC  Growth 
<0 

Time  Period 
(FY) 

PAUC  Growth 
<0 

DSARC 

1970-1980 

5  of  42 

1981-1982 

2  of  6 

Post  Carlucci  DSARC 

1987-1989 

2  of  11 

1983-1986 

10  of  29 

DAB 

1990-1993 

1  of  11 

None 

N/A 

Acquisition  Reform  (AR) 

1994-2000 

4  of  27 

None 

N/A 

DAB  post  AR 

2001-2002 

Oof  6 

2003-2007 

5  of  19 

Negative  PAUC  growth  is  not  regarded  as  a  problem,  probably  eorrectly.  It  is, 
however,  a  elear  and  unexpected  case  in  which  PAUC  growth  reflects  accommodation  to 
the  funding  climate. 

2,  PAUC  Growth  >  50  Percent 

Adoption  of  unrealistically  low  cost  estimates  at  MS  II/B  creates  the  illusion  that  the 
funds  available  over  the  FYDP  and  beyond  will  support  more  MDAPs  than  they  in  fact 
will.  That  is,  unrealistically  optimistic  costing  will  for  a  time  permit  more  new  starts.  In 
addition,  the  conventional  wisdom  holds  that  a  lower  MS  II/B  cost  makes  it  easier  to  gain 
the  concurrence  of  OSD,  the  Office  of  Management  and  Budget,  and  the  Congress  for  a 
new  program.  Consequently,  we  would  expect  to  find  relatively  more  programs  with 
PAUC  growth  of  at  least  50  percent  in  Relatively  Constrained  funding  climates,  which  is 
in  fact  what  we  do  find. 

Table  6  reports  the  number  of  programs  with  an  average  PAUC  growth  of  at  least 
50  percent.  Of  the  fifty-four  programs  that  passed  MS  II/B  in  a  Relatively 
Accommodating  funding  climate,  only  four  showed  PAUC  growth  of  at  least  50  percent. 
In  contrast,  thirty-six  of  the  ninety-seven  programs  that  passed  MS  II/B  in  a  Relatively 
Constrained  funding  climate  showed  cost  growth  of  at  least  50  percent.  This  is  to  say  that 
the  frequency  of  MDAPs  with  a  PAUC  growth  of  at  least  50  percent  is  much  lower  in 
periods  when  the  topline  is  Relatively  Accommodating  than  in  a  Relatively  Constrained 
funding  climate — 7  percent  versus  37  percent. 


It  is  not  clear  that  doing  this  ever  makes  financial  sense  because  the  “loans”  created  by  unrealistically 
low  cost  estimates  eventually  must  be  made  good  one  way  or  another  at  an  implicit  but  steep  interest 
rate. 
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Table  6.  Number  of  PAUC  Growth  Observations  >50% 
by  Acquisition  Regime  and  Funding  Climate 


Topline  Relatively 
Constrained 

Topline  Relatively 
Accommodating 

Acquisition  Regime 

Time  Period 
(FY) 

PAUC  Growth 
>50% 

Time  Period 
(FY) 

PAUC  Growth 
>50% 

DSARC 

1970-1980 

15  of  42 

1981-1982 

Oof  6 

Post  Carlucci  DSARC 

1987-1989 

2  of  11 

1983-1986 

3  of  29 

DAB 

1990-1993 

3  of  11 

None 

N/A 

Acquisition  Reform  (AR) 

1994-2000 

13  of  27 

None 

N/A 

DAB  post  AR 

2001-2002 

3  of  6 

2003-2007 

1  of  19 

3,  Reexamination  of  Trends  in  PAUC  Growth 

The  circumstances  in  which  we  are  more  likely  to  see  very  high  PAUC  growth  and 
instances  of  negative  PAUC  growth  suggest  that  they  reflect  accommodations  to  different 
funding  climates.  In  other  words,  instances  of  high  PAUC  and  negative  PAUC  may  not 
reflect  the  normal  operation  of  the  acquisition  process.  On  this  basis,  Table  7  presents 
average  PAUC  growths  computed  excluding  observations  of  greater  than  or  equal  to  50 
percent  and  negative  values. 

The  statistical  analysis  of  the  data  in  Table  7  produces  the  same  conclusions  as  that 
of  Table  3  in  one  important  respect:  there  is  no  indication  of  statistically  significant 
differences  across  acquisition  regimes  within  a  funding  climate.  The  difference  between 
the  averages  in  the  two  funding  climates  for  the  DAB  post  AR  is,  however,  statistically 
significant  at  the  1  percent  level,  but  the  differences  for  the  other  two  cases  (DSARC  and 
Post-Carlucci  DSARC)  are  not  statistically  different  at  conventional  levels  of 
significance. 


Table  7.  Average  PAUC  Growth  Excluding  Observations  >50  Percent 
and  Negative  Observations  by  Acquisition  Regime  and  Funding  Climate 


Acquisition  Regime 

Topline  Relatively 
Constrained 

Topline  Relatively 
Accommodating 

Time 

Period  (FY) 

%  of  PAUC 
Growth 

Time 

Period 

(FY) 

%  of  PAUC 
Growth 

DSARC 

1970-1980 

21%  (22) 

1981-1982 

22%  (4) 

Post  Cariucci  DSARC 

1987-1989 

22%  (7) 

1983-1986 

13%  (16) 

DAB 

1990-1993 

21%  (7) 

None 

N/A 

Acquisition  Reform  (AR) 

1994-2000 

22%  (10) 

None 

N/A 

DAB  post  AR 

2001-2002 

29%  (3) 

2003-2007 

10%  (13) 

Note:  Numbers  in  parentheses  are  the  number  of  observations. 
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F.  Implications  for  Discussions  of  Acquisition  Reform 

This  paper  points  to  three  implieations  for  a  discussion  of  acquisition  reform.  First, 
the  relevant  context  for  understanding  PAUC  growth  is  the  interface  between  the 
acquisition  process  and  the  resource  allocation  process.  The  crucial  evidence  behind  this 
point  is  the  strong  association  between  funding  climate  and  PAUC  growth.  Resource 
managers  must  think  in  terms  of  a  portfolio  of  programs  across  mission  areas  and 
commodity  types,  and  extending  from  efforts  in  the  technology  base  through  programs 
nearing  the  end  of  production.  When  a  program  is  completed,  it  opens  a  resource  “hole” 
that  programs  emerging  from  EMD  can  occupy.  In  turn,  programs  earlier  in  the 
acquisition  cycle  can  move  forward  as  well.  When  funding  for  acquisition  turns  down, 
these  holes  get  smaller,  or  close  entirely,  or  require  cuts  in  funding  for  ongoing  programs. 
The  alternatives  available  in  this  circumstance  are  all  undesirable — cancellations  of 
programs,  delays  in  new  starts,  stretches,  and  unrealistic  costing.  The  evidence 
summarized  here  suggests  that  it  is  in  this  context  that  high  PAUC  growth  arises. 

Second,  it  seems  unlikely  that  further  changes  in  the  acquisition  process  would  have 
a  major  effect  on  PAUC  growth.  The  research  found  no  evidence  that  acquisition  policy 
and  process  changes  through  the  years  have  produced  sustained  and  significantly  lower  or 
higher  PAUC  growth.  This  does  not  mean  that  the  DAB  process  does  not  provide  a  useful 
discipline  on  acquisition  programs;  moreover,  further  changes  in  acquisition  policy  or 
process  might  be  warranted  for  reasons  of  good  government.  The  evidence  does,  at  a 
minimum,  suggest  that  the  effects  of  changes  in  the  acquisition  process  since  its 
inauguration  in  the  early  1970s  have  not  had  a  dominant  effect  on  PAUC  growth. 

Third,  it  is  difficult  to  see  that  the  cultures  of  the  DoD  acquisition  organizations  are 
a  crucial  obstacle  to  improved  performance  on  cost  growth.  The  key  point  to  note  is  that 
high  PAUC  growth  is  not  persistent,  but  rather  episodic,  and  correlated  with 
environmental  factors  outside  of  the  control  of  the  acquisition  process.  There  is 
remarkably  little  PAUC  growth  in  periods  when  the  funding  is  Relatively 
Accommodating.  It  seems  fair  to  ask  if  it  makes  sense  to  assert  that  an  entrenched  culture 
sometimes  results  in  high  cost  growth  and  other  times  in  low  cost  growth.  Just  how  is  it 
that  the  A  team  takes  the  field  so  quickly  and  quietly  when  the  budgetary  sun  comes  out? 
And  why  even  in  bad  budgetary  weather  do  more  than  half  of  MDAPs  exhibit 
comparatively  modest  PAUC  growth? 
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Appendix  A 
The  Data 


All  of  the  data  used  in  this  study  were  taken  direetly  or  indireetly  from  SARs. 
Broadly,  two  sorts  of  data  are  required.  The  first  is  MS  II/B  baseline  data  for  eaeh 
MDAP,  espeeially  the  PAUC  and  the  inventory  objeetive,  or  total  quantity  to  be  aequired. 
Seeond,  we  need  the  same  data  from  the  final  SAR  filed  by  an  MDAP,  or  for  programs 
still  underway,  the  Current  Estimate  (CE)  from  the  Deeember  2012  SAR.  Our  PAUC 
growth  estimates  refleet  the  aetual  PAUC  (or  the  most  reeent  CE)  normalized  to  the 
MS  II/B  quantity  and  divided  by  the  MS  II/B  PAUC.  They  are,  in  short,  as  elose  as  we 
ean  get  to  aetual  (quantity  normalized)  PAUC  growth. 

The  first  seetion  of  this  appendix  (The  Population  and  the  Sample)  eontains  an 
overview  of  the  sample  of  MDAPs  for  whieh  we  were  able  to  obtain  PAUC  growth 
estimates  and  its  relationship  to  the  eomplete  population  of  MDAPs.  The  seeond  seetion 
of  this  appendix  (PAUC  Growth  Estimates),  beginning  on  page  A-5,  describes  in  detail 
the  PAUC  estimates  used. 

The  Population  and  the  Sample 

The  “stretch  goal”  of  this  paper  was  to  identify  all  MDAPs  that  have  ever  filed  a 
SAR  and  are  at  least  five  years  into  EMD,  and  to  find  a  PAUC  growth  estimate  relative  to 
the  MS  II/B  baseline  for  each  of  these  programs.  As  is  discussed  in  what  follows,  neither 
of  these  objectives  was  fully  achieved. 

Memo  Entries 

This  research  started  with  the  list  of  programs  that  have  filed  at  least  one  SAR 
implicit  in  the  Defense  Acquisition  Management  Information  Retrieval  System 
(DAMIRS)  and  a  set  of  SARs  available  on  an  Office  of  the  Under  Secretary  of  Defense 
(Acquisition,  Technology,  and  Eogistics)  (OUSD(AT&E))  SIPRNet  website.  Together, 
these  provide  SARs  under  345  distinct  labels. 

Not  all  of  these  were  really  distinct  MDAPs,  however: 

•  In  several  instances,  each  of  the  Services  involved  in  a  Joint  program  annually 
filed  separate  SARs,  which,  apart  from  some  administrative  information,  were 
identical  to  each  other. 

•  In  some  instances,  separate  MDAPs  were  merged  into  a  single  MDAP. 
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•  Conversely,  there  were  instanees  in  whieh  a  single  MDAP  was  divided  into  two 
or  more  separate  programs. 

•  The  initial  list  ineluded  a  number  of  instanees  in  whieh  a  single  MDAP  appeared 
(usually  in  different  years)  under  different  names. 

•  A  few  of  the  items  in  the  initial  list  are  not  MDAPs  but  Major  Automated 
Information  System  (MAIS)  programs. 

•  We  found  programs  that  filed  a  SAR  after  passing  MS  I  but  was  eaneelled 
before  passing  MS  II.  We  also  found  a  program  (Patriot  P3I)  that  filed  only  one 
SAR  and  was,  in  the  next  year,  absorbed  into  the  Patriot  Advaneed  Capability-3 
(PAC-3)  program. 

•  Ballistie  missile  defense  programs  appear  on  the  initial  list  under  three  different 
labels.  In  eaeh  case,  the  label  covers  a  set  of  systems,  not  a  single  MDAP. 

•  The  initial  list  included  four  chemical  demilitarization  programs.  These  are 
MDAPs,  but  the  pressures  on  them  and  considerations  that  go  into  funding 
decisions  on  them  mark  them  out  as  significantly  different  from  major  weapon 
system  acquisition  programs. 

Most  of  these  situations  have  a  straightforward  resolution.  Duplicates,  for  example, 
clearly  should  not  be  in  the  main  database,  and  whether  the  chemical  demilitarization 
programs  and  the  various  incarnations  of  strategic  missile  defense  are  included  is  a  matter 
of  judgment.  We  excluded  both  and  also  excluded  MAIS  programs.  Mergers  and 
separations,  in  contrast,  can  be  problematic.  The  guideline  applied  in  these  cases  was  as 
follows:  Enter  each  program  into  the  main  database  only  once  and,  for  mergers  and 
separations,  the  default  option  is  the  program(s)  that  received  MS  II/B  approval. 

The  thirty-six  labels  that  appeared  on  the  initial  list  of  345  and  that  were  not 
included  in  the  database  were  retained  in  a  separate  listing  as  “memo  entries.”  The 
remaining  309  MDAPs  are  the  population  considered  in  this  study.  The  CD  provided 
with  this  paper  (see  the  pocket  on  the  inside  back  cover)  provides  both  the  main  database 
(Table  A-1)  and  the  memo  entries  (Table  A-2). 

Coverage 

We  have  an  estimate  of  PAUC  relative  to  MS  II/B  for  15 1  of  the  309  MDAPs  in  our 
database — a  bit  less  than  half.  Although  accurate,  this  comparison  leaves  out  an 
important  point.  Approximately  seventy-five  of  the  309  MDAPs  were  cancelled  (i.e., 
terminated  before  going  into  production)  or  truncated  (i.e.,  terminated  with  little  or  no 
production  beyond  TRIP).  Of  the  228  MDAPs  that  eventually  progressed  into  production, 
we  have  cost  estimates  for  about  two-thirds. 
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It  would  be  interesting  to  have  a  reasonable  PAUC  growth  estimate  relative  to  the 
MS  II/B  baseline  for  the  programs  that  were  eaneelled  or  truneated,  but  developing  sueh 
estimates  would  require  far  more  resourees  than  were  available  for  this  paper/  The  paper, 
then,  does  not  provide  a  eomprehensive  pieture  of  eost  growth.  Doing  so  was  not  its 
intent.  The  question  asked  was  whether  ehanges  in  aequisition  poliey  and  proeess  over 
time  have  visibly  had  an  influenee  on  PAUC  growth.  We  asked  that  question  for  MDAPs 
that  passed  MS  II/B  as  ACAT  I  programs  and  progressed  into  full  rate  produetion.  The 
question  is  not  explored  for  programs  that  were  eaneelled  or  truneated  or  that  passed 
MS  II/B  as  ACAT  II  or  ACAT  III  programs. 

Table  A-3  presents  data  on  our  eoverage  of  Army,  Navy/USMC,  USAF,  and  Joint 
programs  for  FY  1969-FY  1974,  FY  1975-FY  1988,  FY  1989-FY  2001,  and  FY  2002- 
FY  2007.  Note  that  eoverage  is  defined  as  the  pereentage  of  programs  with  a  PAUC 
growth  estimate  divided  by  the  number  of  programs  that  had  signifieant  produetion — that 
is,  programs  that  were  not  eaneelled  or  truneated. 

For  FY  1969-FY  1974  and  FY  1975-FY  1988,  programs  were  seleeted  by  the 
Offiee  of  Program  Analysis  and  Evaluation  (PA&E)  to  provide  a  representative  eoverage 
of  MDAPs.  It  appears  that  PA&E  was  reasonably  sueeessful  in  doing  so,  although  Joint 
programs  are  distinctly  under-represented  for  EY  1969-EY  1974,  and  USAE  programs 
are  over-represented.  The  situation  is  much  improved  for  EY  1975-EY  1988.  The 
coverage  ratios  range  from  a  low  of  about  63  percent  for  the  Navy/USMC  and  a  high  of 
92  percent  for  Joint  programs. 

The  coverage  ratios  range  from  59  percent  to  94  percent  for  EY  1989-EY  2001  and 
40  percent  to  82  percent  for  EY  2002-EY  2007.  The  programs  in  these  periods  without 
PAUC  growth  estimates  fall  into  two  categories: 

•  Programs  that  passed  MS  II/B  as  ACAT  II  or  ACAT  III  programs  and  later 
became  ACAT  I  programs. 

•  Programs  that  passed  MS  II/B  as  ACAT  I  programs  but  were  subsequently 
reorganized  in  a  way  that  makes  tracking  the  restructured  program  back  to  the 
MS  II/B  baseline  difficult. 


*  It  is  commonly  assumed  that  there  is  a  close  association  between  PAUC  growth  and  program 
cancellation  or  truncation.  As  a  careful  look  at  the  data  will  show,  high  PAUC  growth  is  neither  a 
necessary  nor  a  sufficient  condition  for  an  MDAP  to  be  cancelled  or  terminated.  Space-Based  Infrared 
System  (SBIRS)-High,  for  example,  had  an  exceptionally  high  PAUC  growth,  but  as  of  the  December 
2012  SAR,  its  planned  acquisition  quantity  remained  at  80  percent  of  the  Milestone  II  baseline.  In 
contrast,  the  Joint  Standoff  Weapon  (JSOW)  PAUC  normalized  to  the  MS  II  quantity  decreased  by  14 
percent,  yet  (again  as  of  the  December  2012  SAR)  the  planned  acquisition  quantity  was  only  38  percent 
of  the  MS  II  baseline. 


A-3 


The  second  of  these  problems  could  be  overcome  at  least  in  some  cases  with  sufficient 
effort,  but  the  first  probably  cannot,  because  the  data  required  cannot  readily  be  obtained 
and,  in  fact,  may  no  longer  exist. 


Table  A-3.  PAUC  Growth  Estimates  as  a  Percentage  of  the  Number  of  MDAPs  for  Which  a 

PAUC  Growth  Estimate  Could  Be  Made 


FY  1969- 
FY1974 

FY1975- 
FY  1988 

FY1989- 
FY  2001 

FY  2002- 
FY  2007 

Army 

MDAPs 

15 

30 

23 

8 

Cancellations  and 
truncations 

4 

7 

7 

3 

Net:  PAUC  growth 
estimate  feasible 

11 

23 

16 

5 

Number  of  MDAPs  with 
PAUC  growth  estimate 

4 

18 

15 

2 

Coverage 

36% 

78% 

94% 

40% 

USN/USMC 

MDAPs 

23 

45 

25 

14 

Cancellations  and 
truncations 

1 

10 

3 

3 

Net:  PAUC  growth 
estimate  feasible 

22 

35 

22 

11 

Number  of  MDAPs  with 
PAUC  growth  estimate 

6 

22 

13 

9 

Coverage 

27% 

63% 

59% 

82% 

USAF 

MDAPs 

16 

31 

20 

10 

Cancellations  and 
truncations 

1 

9 

5 

3 

Net:  PAUC  growth 
estimate  feasible 

15 

22 

15 

7 

Number  of  MDAPs  with 
PAUC  growth  estimate 

6 

15 

12 

5 

Coverage 

40% 

68% 

80% 

71% 

Joint 

MDAPs 

6 

17 

12 

12 

Cancellations  and 
truncations 

2 

5 

4 

5 

Net:  PAUC  growth 
estimate  feasible 

4 

12 

8 

7 

Number  of  MDAPs  with 
PAUC  growth  estimate 

1 

11 

7 

5 

Coverage 

25% 

92% 

88% 

71% 
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PAUC  Growth  Estimates 


As  was  noted  above,  this  study  obtained  or  made  estimates  of  PAUC  growth  for  151 
MDAPs  that  passed  MS  11/B  as  AC  AT  1  programs  during  the  years  FY  1969-FY  2007. 
These  different  sourees  and  methods  are  deseribed  in  what  follows. 

PA&E  Cost  Growth  Database  (PA&E) 

PA&E,  now  the  Office  of  Cost  Analysis  and  Program  Evaluation,  developed  a 
database  of  cost  growth  experienced  by  MDAPs.  This  database  is  documented  in  a 
briefing  by  John  McCrillis  given  at  the  2003  Annual  DoD  Cost  Analysis  Symposium. 
(The  CD  included  with  this  paper  includes  this  briefing.) 

The  earliest  cost  estimates  in  the  PA&E  database  are  from  MDAPs  that  passed  MS 
11  in  1970,  and  the  latest  are  for  programs  that  passed  MS  B  in  EY  2001.  The  MDAPs  in 
the  PA&E  database  were  last  updated  using  the  December  2004  SARs.  The  PA&E  cost 
growth  database  included  PAUC  growth  estimates  for  93  MDAPs  that  were  completed 
(that  is,  filed  their  last  SAR)  in  EY  2004  or  earlier.  The  other  PAUC  growth  estimates  in 
the  PA&E  database  had  to  be  updated — ^which  we  did  not  have  the  data  or  resources  to 
do — or  replaced  with  a  PAUC  growth  estimate  made  in  some  more  summary  fashion. 

No  Quantity  Change  (NQC) 

The  CE  quantity  was  within  ±  1  percent  of  the  MS  11/B  quantity  for  thirteen  of  the 
MDAPs  from  EY  1989-EY  2007.  No  quantity  normalization  is  needed  for  these 
programs;  their  PAUC  growth  is  computed  by  dividing  the  CE  PAUC  in  the  final  SAR 
(or  the  December  2012  SAR  for  an  ongoing  program)  by  the  MS  11/B  PAUC  and 
subtracting  1.  The  PAUC  growth  for  SBIRS-High  (H)  also  falls  under  this  heading.  The 
total  number  of  SBIRS-H  satellites  to  be  acquired  decreased  from  five  (at  MS  11)  to  four 
(the  December  2012  SAR).  The  decrease,  however,  was  in  a  satellite  purchased  with 
Research,  Development,  Test  and  Evaluation  (RDT&E)  funds,  and  we  did  not  put  these 
on  a  learning  curve.  There  was  no  change  in  the  number  of  SBlRS-H  satellites  purchased 
with  procurement  funds.  Einally,  although  the  PAC-3  quantity  change  fell  outside  the  ±  1 
percent  boundary,  data  limitations  made  it  necessary  to  compute  the  PAC-3  PAUC 
growth  as  the  ratio  of  the  CE  and  MS  11  PAUCs.  The  relevant  data  for  the  NQC 
computations  are  included  on  the  NQC  worksheet  in  Table  A-4  on  the  CD.  The  letters  in 
the  computation  box  at  the  top  of  the  worksheet  refer  to  the  column  headings  for  the  data. 


Accessible  at  https://dodcas.deltaresources.eom/DODCAS%20Archives 

/36th%20DODCAS%20%282003%29/Theme%20-%20Evolutionary%20Acquisition/McCrillis.pdf. 
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DAMIRS  Learning  Curves  (DLC) 

The  DoD  contractor  staff  for  DAMIRS  provided  us  with  their  estimates  of  learning 
curve  parameters  that  we  were  able  to  use  to  compute  PAUC  growth  for  fourteen  MDAPs 
that  passed  MS  II/B  during  FY  1989-FY  2001.  We  refer  to  these  as  the  DAMIRS 
Learning  Curve  (DLC)  PAUC  growth  estimates.  For  each  of  these,  we  took  the  CE 
PAUC  growth  in  program  base-year  dollars  from  the  last  SAR  for  the  program  or  the 
December  2012  SAR  (for  still  ongoing  programs).  The  task  was  to  normalize  this  PAUC 
estimate  to  the  MS  II/B  quantity,  which  was  done  as  follows: 

•  We  used  the  learning  curve  to  compute  the  recurring  flyaway  cost  at  the  MS  II/B 
baseline  quantity. 

•  The  CE  estimates  of  RDT&E  and  non-recurring  flyaway  cost  were  taken  from 
the  final  SAR  for  the  program  or  from  the  December  2012  SAR  (for  still  on¬ 
going  programs). 

•  Support  costs  paid  for  with  procurement  dollars  are,  for  many  programs, 
primarily  initial  spares  and  support  equipment,  although  other  items  may  also 
fall  into  this  category.  Initial  spares  and  support  equipment  normally  scale  with 
the  number  of  units  of  the  system  purchased.  Eor  that  reason,  we  used  CE 
support  cost  reported  in  the  last  or  most  recent  SAR  scaled  to  the  MS  lEB 
baseline  quantity. 

The  computations  and  the  data  used  are  laid  out  on  the  DEC  worksheet  in  Table  A-4  on 
the  CD  provided  with  this  paper.  The  letters  in  the  computation  box  at  the  top  of  the 
worksheet  refer  to  the  column  headings  for  the  data. 

Calibrated  Learning  Curve  (CLC) 

There  were  twenty-nine  MDAPs  for  which  we  did  not  have  a  PA&E  estimate  and 
did  not  have  estimated  learning  curve  parameters,  and  for  which  the  CE  quantity  was 
significantly  different  from  the  MS  II/B  quantity.  The  approach  we  used  in  those  cases 
rested  on  a  cost  progress  curve  of  the  conventional  form: 

C  =  TQl^  (1) 

In  this  expression,  C  is  recurring  flyaway  cost,  T  is  first  unit  cost,  Q  is  cumulative 
production,  and  P  is  the  cost  progress  parameter.  We  solved  this  and  used  the  CE  for 
recurring  flyaway  to  get: 

f  =  CQ-I^  (2) 

A  value  of  P  =  0.94  was  used  for  each  of  the  programs.  This  will  be  referred  to  as  the 
calibrated  learning  curve  (CEC)  method.  Erom  this  point,  the  computations  were  the 
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same  as  those  for  MDAPs  for  whieh  DAMIRS  staff  provided  the  learning  eurve 
parameters. 

The  CLC  worksheet  in  Table  A-4  on  the  CD  provides  the  data  used  in  making  the 
eomputations  and  indieates  the  details  of  the  eomputations.  The  letters  in  the  eomputation 
box  at  the  top  of  the  worksheet  refer  to  the  eolumn  headings  for  the  data. 

Summary 

Table  A-5  below  provides  a  summary  of  the  sourees  of  the  PAUC  growth  estimates 
for  three  different  periods  (whieh  were  marked  off  only  for  eonvenienee  in  assembling 
the  data).  Table  A-1  on  the  CD  identifies  the  souree  of  eaeh  estimate. 


Table  A-5.  Sources  of  the  PAUC  Growth  Estimates  Used  in  Different  Periods 


Period  (FY) 

PA&E 

NQC* 

DLC 

CLC 

Total 

1970-1988 

83 

0 

0 

0 

83 

1989-2001 

10 

6 

14 

17 

47 

2002-2007 

0 

8 

0 

13 

21 

Total 

93 

14 

14 

30 

151 

*  No  Quantity  Change  (i.e.,  CE  quantity  with  ±  1  percent  of  the  MS  ll/B  quantity.) 


Comparison  of  the  PA&E  and  CLC  PAUC  Growth  Estimates 

The  data  in  Table  A-5  suggest  the  question  of  whether  the  results  are  infiueneed 
signifieantly  by  the  different  ways  in  whieh  PAUC  estimates  are  made.  This  is  primarily 
a  question  about  FY  2002-FY  2007,  whieh  has  a  notably  low  average  PAUC  growth  (10 
pereent)  and  for  whieh  about  two-thirds  of  the  PAUC  growth  estimates  were  made  with 
the  CLC  method.  It  is  seeondarily  a  question  about  FY  1989-FY  2001  in  relation  to 
FY  1970-FY  1988.  In  the  earlier  period,  all  of  the  PAUC  growth  estimates  were  from 
PA&E,  while  in  the  later,  about  two-thirds  were  made  using  either  the  DEC  or  the  CEC 
method. 

The  obvious  approaeh  to  this  issue  is  to  eompare  the  PA&E  PAUC  growth  for 
systems  that  have  been  eompleted  with  PAUC  growth  for  those  same  systems  eomputed 
using  the  DEC  and  the  CEC  methods.  Unfortunately,  there  are  no  MDAPs  that  have  been 
eompleted  and  for  whieh  we  have  both  a  PA&E  PAUC  growth  estimate  and  the  data 
needed  to  eompute  a  DEC  or  an  CEC  estimate. 

The  best  we  ean  do  is  to  examine  the  twenty-three  MDAPs  that  passed  MS  II/B 
during  EY  1989-EY  2001  and  for  whieh  we  have  a  PA&E  PAUC  growth  estimate,  a 
DEC  estimate,  and  a  CEC  estimate.  As  was  noted  above,  the  PA&E  estimates  were  most 
reeently  updated  with  the  2004  SARs.  The  DEC  and  CEC  estimates,  in  eontrast, 
ineorporated  more  reeent  data — either  the  final  SAR  for  the  program  or,  for  ongoing 
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programs,  the  December  2012  SAR.  Consequently,  in  most  cases  we  would  expect  the 
DLC  and  CLC  PAUC  growth  estimates  to  be  larger  than  the  corresponding  PA&E 
estimate.  That  is  the  test:  A  method  fails  if  it  yields  estimates  that  are  “too  often”  and  by 
“too  much”  less  than  the  PA&E  estimates.  Clearly,  this  is  a  weak  test. 

The  relevant  estimates  are  presented  in  Table  A-6  on  page  A-9.  The  comparison  of 
the  PA&E  estimates  and  CEC  estimates  is  on  the  left,  and  the  comparison  of  the  PA&E 
and  DEC  estimates  is  on  the  right.  The  CEC  estimates  are  larger  than  the  PA&E 
estimates  for  seventeen  of  the  twenty-three  MDAPs — in  most  cases,  considerably  larger. 
They  are  smaller  in  six  cases  (shaded  rows).  In  all  but  one  of  these  cases  (Joint  Direct 
Attack  Munition,  or  JDAM)  the  differences  are  absolutely  or  relatively  small.  The 
average  of  CEC  PAUC  growth  estimates  is  77  percent  in  comparison  to  an  average  of  60 
percent  for  the  PA&E  estimates.  The  DEC  estimates  exhibit  the  same  pattern.  The 
average  of  the  DEC  estimates  is  73  percent,  and  four  of  them  (shaded  rows)  are  less  than 
the  PA&E  estimate  for  the  program,  three  by  a  substantial  amount. 
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Table  A-6.  Comparison  of  PA&E,  CLC,  and  DLC  PAUC  Growth  Estimates  for  23  MDAPs 


Program 

PA&E 

CLC 

Program 

PA&E 

DLC 

LONGBOW  APACHE 

78% 

117% 

LONGBOW  APACHE 

78% 

133% 

F-22 

41% 

71% 

F-22 

41% 

55% 

F/A-18E/F 

6% 

12% 

F/A-18E/F 

6% 

9% 

BRADLEY  UPGRADE 

39% 

54% 

BRADLEY  UPGRADE 

39% 

86% 

MIDS 

30% 

72% 

MIDS 

30% 

68% 

CEO 

48% 

62% 

CEO 

48% 

62% 

H-1  UPGRADES 

124% 

192% 

H-1  UPGRADES 

124% 

197% 

LPD  17 

43% 

71% 

LPD  17 

43% 

72% 

CH-47F 

147% 

173% 

CH-47F 

147% 

156% 

GMLRS/GMLRS  AW 

125% 

249% 

GMLRS/GMLRS  AW 

125% 

243% 

MH-60S 

62% 

69% 

MH-60S 

62% 

70% 

Tactical  Tomahawk 

24% 

28% 

Tactical  Tomahawk 

24% 

27% 

GBS 

10% 

31% 

GBS 

10% 

33% 

Stryker 

21% 

25% 

Stryker 

21% 

22% 

UH-60M  Black  Hawk 

49% 

62% 

UH-60M  Black  Hawk 

49% 

61% 

WGS 

28% 

55% 

WGS 

28% 

42% 

C-130J 

70% 

84% 

C-130J 

70% 

70% 

JPATS 

43% 

40% 

JPATS 

43% 

44% 

SSN  774 

35% 

33% 

SSN  774 

35% 

37% 

JDAM 

18% 

-10% 

JDAM 

18% 

-13% 

JAVELIN 

229% 

197% 

JAVELIN 

229% 

134% 

MH-60R 

95% 

74% 

MH-60R 

95% 

80% 

NAS 

25% 

21% 

NAS 

25% 

1% 

Average 

60% 

77% 

60% 

73% 

Note:  The  PA&E  estimates  were  updated  only  through  the  2004  SARs.  The  CLC  and  DLC  estimates 
incorporate  information  from  the  last  SAR  for  the  program  or  the  December  2012  SAR  (for  still  ongoing 
programs). 
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Appendix  B. 

Evidence  on  the  Influence  of  the  Funding  Climate 
Prevailing  at  MS  II/B  on  PAUC  Growth 


As  was  noted  in  the  main  body  of  this  paper  (page  4),  the  PAUC  growth  of  eaeh 
program  is  assigned  to  the  aequisition  regime/funding  elimate  in  place  when  the  program 
passed  MS  II/B  or  filed  its  first  SAR.  The  extent  to  which  this  is  reasonable  is  not 
immediately  clear.  On  the  one  hand,  a  program  can  easily  take  ten  or  fifteen  years  from 
the  start  of  EMD  through  delivery  of  the  final  production  lot  and  thus  spend  part  of  its 
acquisition  cycle  under  successive  acquisition  regimes.  On  the  other  hand,  our  estimates 
are  of  PAUC  growth  measured  from  the  baseline  established  at  MS  II/B.  This  does  not 
change  over  the  course  of  a  program’s  acquisition  cycle,  and  to  the  extent  that  the  main 
causes  of  cost  growth  are  “baked  into”  the  MS  II/B  baseline,  we  would  not  expect 
subsequent  changes  in  funding  climate  or  acquisition  regime  to  have  a  major  effect  on 
PAUC  growth. 

This  appendix  presents  evidence  that  bears  on  whether  the  actual  acquisition  costs 
of  a  program  are  significantly  influenced  by  policy  or  process  changes  made  after  its 
MS  II/B. 

Case  I  in  Table  B-1  compares  the  FY  1970-FY  1972  cohort  with  that  of  FY  1978- 
FY  1980.  A  program  that  passed  MS  II  in  1970  was  eleven  years  away  from  the  next 
boom  period  of  defense  spending,  which  began  in  FY  1981.  In  contrast,  a  program  that 
passed  MS  II  in  1980  was  born  at  the  boundary  of  a  period  of  growing  funding;  most  of 
these  programs  would  be  expected  to  have  completed  FMD  and  entered  FRIP  before  the 
Carter-Reagan  defense  boom  ended  with  the  FY  1986  budget.  We  might  then  expect  the 
FY  1978-FY  1980  cohort  to  have  a  lower  average  PAUC  growth  associated  with  a 
Relatively  Accommodating  funding  climate.  In  fact,  we  find  the  opposite.  Similarly,  in 
Case  2,  we  see  the  average  PAUC  growth  for  the  FY  2000-FY  2002  cohort  is  the  same  as 
that  for  the  FY  1987-FY  1989  cohort,  even  though  starting  in  FY  2003,  DoD  entered  a 
Relatively  Accommodating  funding  climate.  In  Case  3  we  find  average  PAUC  growth 
higher  for  FY  I984-FY  1986  than  for  FY  I98I-FY  1982  but  still  at  a  low  value 
characteristic  of  periods  of  a  Relatively  Accommodating  funding  climate.  The  data  then 
favor  the  position  that  the  PAUC  growth  of  programs  tends  to  be  characteristic  of  those 
of  the  periods  in  which  they  enter  FMD  and  a  complete  Acquisition  Program  Baseline  for 
the  program  is  first  established. 
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Table  B-1.  Comparison  of  First  Three  and  Last  Three  Years  of  Three  Funding  Climates  in 
Terms  of  Average  PAUC  Growth  and  Instances  of  PAUC  Growth  of  at  Least  50% 

Case  1:  Next  Period  of  Relatively  Accommodating  Topline  FY1979-FY1986 


Passed  MS  II  during 

Average  PAUC 
Growth 

PAUC  Growth  >50%  No.  of  Observations 

1970-1972 

18% 

2 

9 

1978-1980 

24% 

2 

9 

Case  2:  Next  Period  of  Relatively  Accommodating  Topline  FY2002-FY2007 

Passed  MS  II  during 

Average  PAUC 
Growth 

PAUC  Growth  >50%  No.  of  Observations 

1987-1989 

34% 

2 

11 

2000-2002 

45% 

3 

8 

Case  3:  Next  Period  of  Relatively  Constrained  Topline  FY1987-FY2001 

Passed  MS  II  during 

Average  PAUC 
Growth 

PAUC  Growth  >50%  No.  of  Observations 

1981-1983 

4% 

0 

19 

1984-1986 

22% 

3 

16 
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Appendix  C. 

Average  PAUC  Growth  by  Acquisition  Regime, 
Topline  Condition,  and  Military  Department/ 

Joint  Program 


Table  C-1.  MDAPs  Average  PAUC  Growth  by  Service,  Acquisition  Regime,  and  Topline 

Condition 


Service/ 

Acquisition  Regime 

Topline  Relatively  Constrained 

Topline  Relatively  Accommodating 

Period  (FY) 

PAUC  Growth 

Period  (FY) 

PAUC  Growth 

ARMY 

DSARC 

1970-1980 

36%  (13) 

1981-1982 

N/A 

DSARC  post  Carlucci 

1987-1989 

48%  (6) 

1983-1986 

10%  (6) 

DAB 

1990-1993 

42%  (5) 

None 

N/A 

Acquisition  Reform  (AR) 

1994-2000 

82%  (6) 

None 

N/A 

DAB  post  AR 

2001-2002 

62%  (1) 

2003-2007 

29%  (2) 

NAVY 

DSARC 

1970-1980 

25%  (13) 

1981-1982 

-7%  (1) 

DSARC  post  Cariucci 

1987-1989 

17%  (2) 

1983-1986 

10%  (12) 

DAB 

1990-1993 

31%  (3) 

None 

N/A 

Acquisition  Reform  (AR) 

1994-2000 

55%  (9) 

None 

N/A 

DAB  post  AR 

2001-2002 

9%  (1) 

2003-2007 

8%  (9) 

USAF 

DSARC 

1970-1980 

38%  (10) 

1981-1982 

8%  (4) 

DSARC  post  Cariucci 

1987-1989 

11%  (1) 

1983-1986 

6%  (7) 

DAB 

1990-1993 

49%  (2) 

None 

N/A 

Acquisition  Reform  (AR) 

1994-2000 

77%  (7) 

None 

N/A 

DAB  post  AR 

2001-2002 

75%  (3) 

2003-2007 

-7%  (4) 

JOINT 

DSARC 

1970-1980 

47%  (6) 

1981-1982 

37%  (1) 

DSARC  post  Cariucci 

1987-1989 

20%  (2) 

1983-1986 

38%  (4) 

DAB 

1990-1993 

-7%  (1) 

None 

N/A 

Acquisition  Reform  (AR) 

1994-2000 

50%  (5) 

None 

N/A 

DAB  post  AR 

2001-2002 

45%  (1) 

2003-2007 

8%  (4) 

Note:  Numbers  in  parentheses  are  the  number  of  observations  available. 
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